Cognitive replication through augmented reality

ABSTRACT

In one embodiment of the present invention, environment information corresponding to a user is received. A target environment is simulated in augmented reality based on the environment information. Physiological information corresponding to the user is received. A cognitive state of the user is determined based on the physiological information. In response to determining that the cognitive state of the user is not a target cognitive state, the target environment is modified in augmented reality to achieve the target cognitive state.

BACKGROUND OF THE INVENTION

The present invention relates generally to the field of augmentedreality, and more particularly to cognitive replication throughaugmented reality.

Augmented reality (AR) is a live direct or indirect view of a physical,real-world environment whose elements are augmented (or supplemented) bycomputer-generated sensory input such as sound, graphics, or GPSinformation. AR is related to computer-mediated reality, a more generalconcept in which a view of reality is modified (possibly even diminishedrather than augmented) by a computer. AR technology functions byenhancing the current perception of reality for a user.

Through AR technology, information about the surrounding real worldenvironment of a user becomes interactive and digitally manipulable.Information about the virtual environment and its elements is overlaidon the real world. This information can be virtual or real, e.g., seeingother real sensed or measured information such as electromagnetic radiowaves overlaid in exact alignment with where they actually are in space.Augmented reality brings out the components of the digital world into aperceived real world of the user.

SUMMARY

In one embodiment of the present invention, environment informationcorresponding to a user is received, wherein the environment informationcomprises information selected from the group consisting of: trafficdensity during a commute of the user to the athletic competitionsetting, a commute time of the user to the athletic competition setting,and weather at the athletic competition setting. A target environment issimulated in augmented reality based on the environment information.Physiological information corresponding to the user is received, whereinthe physiological information comprises information selected from thegroup consisting of: brain wave information, heart rate, breath rate,internal body temperature, and blood pressure. A cognitive state of theuser is determined based on the physiological information. In responseto determining that the cognitive state of the user is not a targetcognitive state, the target environment is modified in augmented realityto achieve the target cognitive state, wherein modifying the targetenvironment comprises: simulating, in augmented reality, objects orsounds in the target environment, and wherein objects include fanavatars or opponent avatars, and wherein sounds include crowd noise oropponent taunts. In response to determining that the cognitive state ofthe user is the target cognitive state, evaluating a user performancewhile the user is in the target cognitive state. In response todetermining that the modification achieves the target cognitive state,storing the modification as an indicator of the cognitive state for theuser.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram depicting one example of anaugmented reality learning environment, in accordance with one or moreembodiments disclosed herein.

FIG. 2 a flowchart depicting one example of a method for cognitivealteration through augment reality, in accordance with one or moreembodiments disclosed herein.

FIG. 3 is a block diagram of components of a computing system, inaccordance with one or more embodiments disclosed herein.

DETAILED DESCRIPTION

The present invention will now be described in detail with reference tothe Figures. The descriptions of various instances, scenarios, andexamples related to the present invention are presented for purposes ofillustration and are not intended to be exhaustive or limited to theembodiments disclosed.

FIG. 1 is a plan view illustration depicting one example of augmentedreality learning environment 100, in accordance with one or moreembodiments disclosed herein. FIG. 1 provides only an illustration ofone implementation and does not imply any limitations with regard to theenvironments in which different embodiments may be implemented. Somemodifications to the depicted environment may be made by those skilledin the art without departing from the scope of the invention as recitedby the claims. As depicted, learning environment 100 includes network102, computing device 110, learning program 112, data repository 114,and augmented reality (AR) device 120.

In some embodiments, computing device 110 and AR device 120 areinterconnected and communicate through network 102. In some embodiments,network 102 is a local area network (LAN), a telecommunications network,a wireless local area network (WLAN), such as an intranet, a wide areanetwork (WAN), such as the Internet, or any combination thereof, and caninclude wired, wireless, or fiber optic connections. In someembodiments, network 102 can generally be any combination of connectionsand protocols that will support communications between computing device110, AR device 120, and any other computing device (not shown) connectedto network 102.

In some embodiments, computing device 110 is any electronic device, orcombination of electronic devices, capable of executing computerreadable program instructions and communicating with any computingdevice within computing environment 100. For example, computing device110 may be a workstation, personal computer, laptop computer, tablet,personal digital assistant, or mobile phone. In some embodiments,computing device 110 is a computer system utilizing clustered computersand components (e.g., database server computers, application servercomputers) that act as a single pool of seamless resources when accessedby elements of computing environment 100. For example, computing device110 may be a data center in a cloud computing environment. In someembodiments, computing device 110 includes components as depicted anddescribed with respect to computing system 300 in FIG. 3.

In some embodiments, learning program 112 is any computer program,application, subprogram of a larger program, such as an OS, or acombination thereof that determines environmental elements for AR device120 to provide a user in order to achieve a target cognitive state ofthe user.

In some embodiments, learning program 112 determines a cognitive stateof a user of AR device 120 based on the physiological informationcollected by AR device 120. A cognitive state, for example, may be anymental state of the user such as stressed or relaxed. Learning program112 can additionally determine a degree of the cognitive state. Forexample, learning program 112 may determine that a user is in a highstress state or a low stress state.

As depicted in FIG. 1, learning program 112 is located in computingdevice 110. In other embodiments, learning program 112 is located in ARdevice 120 or any other device connected to network 102.

In some embodiments, data repository 114 contains environmentinformation and physiological information collected by AR device 120. Insome embodiments, data repository 114 stores the environment informationand cognitive information collected by AR device 120.

In some embodiments, data repository 114 can be implemented with anon-volatile storage media known in the art. For example, directorydatabase 114 may be implemented with a tape library, optical library,one or more independent hard disk drives, or multiple hard disk drivesin a redundant array of independent disks (RAID). In an embodiment,directory database 114 can be implemented using a suitable storagearchitecture known in the art. For example, directory database 114 maybe implemented as a relational database, an object-oriented database, oran object-relational database.

In some embodiments, AR device 120 is a computing device that provides aview, direct or indirect, of a real-world environment with environmentalelements that are computer-generated. In some embodiments, environmentalelements include visual and/or audial elements. In some embodiments, ARdevice 120 can be a wearable device or a hand-held device.

In some embodiments, AR device 120 includes one or more sensors thatcollecting environment information and physiological information.Environment information may include, for example, social interactions ofa user of AR device 120, traffic information (e.g., traffic densityduring a commute of the user, a commute time of the user), weather atthe location of the user, and any elements that cause stress for theuser. Physiological information may include brain wave information of auser of AR device 120, heart rate of the user, breathing rate of theuser, internal body temperature of the user, and blood pressure of theuser. In some embodiments, AR device 120 includes components as depictedand described with respect to computing system 300.

FIG. 2 is a flowchart of method 200 depicting one example of operationalsteps for cognitive alteration through augment reality, in accordancewith one or more embodiments disclosed herein. In some embodiments,workflow 200 is performed by learning program 112. In other embodiments,workflow 200 is performed by any other computer program while workingwith learning program 112. In some embodiments, learning program 112begins performing workflow 200 in response to receiving an indication bya user of AR device 120 or any other computing device connected tonetwork 102.

Learning program 112 receives (205) target environment information. Atarget environment may be a setting to be simulated by AR device 120 fora user. In some embodiments, the target environment is a vocationalsetting of the user. For example, in an instance where the user is anathlete, the target environment may be a corresponding athleticcompetition. As another example, in an instance where the user is asurgeon, the target environment may be an operating room.

In some embodiments, learning program 112 receives target environmentinformation from AR device 120. AR device 120 may collect the targetenvironment information through one or more sensors. In someembodiments, target environment information is any objects or soundsrelated to the target environment. Target environment information mayinclude social interactions of a user of AR device 120, trafficinformation (e.g., traffic density during a commute of the user, acommute time of the user), weather at the location of the user, and anyelements that cause stress for the user. In some embodiments, targetenvironment information is stored in data repository 114.

An as example, a user may be a golfer, and the target environment may bea golf tournament. Learning program 112 may collect target environmentinformation throughout the day of the golf tournament. For example,target environment information collected may include social interactionsbetween the user and others prior to the golf tournament (e.g., amotivational speech from a friend, instruction from a coach, etc.) aswell as social interactions during the tournament (e.g., a loud cheerfrom a fan during a backswing of the user, taunts from opponents whileputting, etc.). Learning program 112 may further collect informationrelated to weather conditions (e.g., foggy and cold weather, time ofsunset, etc.) and the natural environment (e.g., a flock of geese flyingover the course, etc.) during the golf tournament. Learning program 112may similarly collect target environment information from multiple golftournaments played by the user and store the collected information indata repository 114.

Learning program 112 simulates (210) the target environment. In someembodiments, learning program 112 simulates, in augmented reality (AR),the target environment based on the target environment informationcollected and stored. Learning program 112 may determine which targetenvironment information has been collected repeatedly, or most often,and simulate those elements in AR through AR device 120.

Continuing the golf example, learning program 112 may determine that theuser often receives instruction from a coach immediately prior to takingthe first swing at each hole and that a fan often holds up a largeyellow sign in front of the user throughout the course of play. During apractice session of golf while the user is using AR device 120, learningprogram 112 may then simulate, through AR device 120, the instructionfrom the coach as an audial element and the sign from the fan as avisual element at their appropriate times while the user is playing thepractice session.

Learning program 112 determines (215) if a target cognitive state isachieved. In some embodiments, learning program 112 determines thecognitive state of a user and compares the determined cognitive stateagainst the target cognitive state. The cognitive state of the user maybe any mental state such as stressed, relaxed, excited, or angry.Further, the cognitive state may be a degree of a mental state such as ahigh stress state or a low stress state. As an example, in an instancewhere learning program 112 determines that the cognitive state of a useris a relaxed state, and the target cognitive state is a stressed state,learning program 112 determines that target cognitive state is notachieved.

In some embodiments, learning program 112 determines the cognitive stateof the user based on biometric information corresponding to the user.Biometric information may include, for example, physiologicalmeasurements such as brain wave information, heart rate, respiratoryrate, internal body temperature, and blood pressure. In someembodiments, AR device 120 receives the biometric informationcorresponding to the user of AR device 120 through one or more sensors.

If the determined cognitive state matches the target cognitive state(decision block 215, YES branch), learning program 112 maintains thesimulated environment.

If the determined cognitive state does not match the target cognitivestate (decision block 215, NO branch), learning program 112 modifies(220) the simulated environment (i.e., the augmented realityenvironment) in order to achieve the target cognitive state. Forexample, in an instance where learning program 112 determines that thecognitive state of a user is a relaxed state, and the target cognitivestate is a stressed state, learning program 112 modifies the simulatedenvironment in a manner that increases the stress of the user.

Continuing the golf example, the cognitive state of the user may bemonitored throughout the practice session of golf. The target cognitivestate of the user may be a high stress state. During the first two holesof the practice session, the cognitive state of the user is determinedto be a high stress state. Therefore, the target cognitive state isachieved and the simulated environment and elements are maintained,unchanged, during that time. However, while the user is putting on thethird hole during the practice session, the cognitive state of the useris determined to be relaxed, despite, e.g., the presence of thesimulated fan holding a large yellow sign prior to, and during, theputt. Therefore, the target cognitive state is not achieved at thatinstance, and the simulated environment will be modified in order toachieve the high stress target cognitive state while the user isputting.

In some embodiments, learning program 112 modifies various elements ofthe AR simulated environment such as visual elements (i.e., objects) andaudial elements (i.e., sounds). Simulated sounds may be binaudial, i.e.,relating to both ears of a user. Modifications may include, for example,increasing the size or quantity of a currently simulated element.Modifications may further include adding a new visual or audial elementor removing a currently simulated element. In instances where the targetenvironment is an athletic competition, simulated objects may include,for example, fan avatars, signs, and opposing player avatars. Simulatedsounds may include, for example, crowd noise, white noise, shouts fromfans, taunts from opposing players, and impaired (e.g., lower inloudness, fewer in number) communications from teammates.

Continuing the golf example, to increase the stress of the user whileputting, learning program 112 determines, based on the targetenvironment information stored in data repository 114, that a flock ofgeese occasionally flies over the golf course while the user is puttingand that a fan often shouts the name of the user throughout the courseof play. These visual and audial elements are therefore simulated duringputts in order to increase the stress of the user.

In some embodiments, the simulated environment is modified based onsuccessful modifications previously made for that user or other users. Asuccessful modification may be a modification that caused the cognitivestate of a user to be closer to the target cognitive state. Learningprogram 112 may determine if there is such success information forsimilar users stored in data repository 114. A similar user may be, forexample, a user with a target environment or target cognitive statesimilar to that of the current user. If there is a similar user,augmented reality elements of the simulated environment may be modifiedsimilarly to the previous user.

In some embodiments, learning program 112 monitors a task performance ofa user while monitoring the cognitive state of the user. Further,learning program 112 may determine whether the task performance whilethe user is in the cognitive state as compared with the task performancewhile the user is not in the cognitive state. A task may be, forexample, an athletic competition where the user is an athlete or avocational duty of the user such as putting out a fire where the user isa firefighter.

Continuing the golf example, learning program 112 may monitor how wellthe user plays throughout the course of the practice session of golfwhile the user is in a high stress state and while the user is not in ahigh stress state.

FIG. 3 depicts computing system 300, which illustrates components ofcomputing device 110 and client device 120. Computing system 300includes processor(s) 301, cache 303, memory 302, persistent storage305, communications unit 307, I/O interface(s) 306, and communicationsfabric 304.

Communications fabric 304 provides communications between cache 303,memory 302, persistent storage 305, communications unit 307, and I/Ointerface(s) 306. Communications fabric 304 can be implemented with anyarchitecture designed for passing data and/or control informationbetween processors (e.g., microprocessors, communications and networkprocessors, etc.), system memory, peripheral devices, and any otherhardware components within a system. For example, communications fabric304 may be implemented with one or more buses or a crossbar switch.

Memory 302 and persistent storage 305 are computer readable storagemedia. In some embodiments, memory 302 includes random access memory(RAM) (not shown). In general, memory 302 may include any suitablevolatile or non-volatile computer readable storage media. Cache 303 is afast memory that enhances the performance of processors 301 by holdingrecently accessed data, and data near recently accessed data, frommemory 302.

Program instructions and data used to practice embodiments of thepresent invention may be stored in persistent storage 305 and in memory302 for execution by one or more of the respective processors 301 viacache 303. In some embodiments, persistent storage 305 includes amagnetic hard disk drive. Alternatively, or in addition to a magnetichard disk drive, persistent storage 305 may include a solid state harddrive, a semiconductor storage device, read-only memory (ROM), erasableprogrammable read-only memory (EPROM), flash memory, or any othercomputer readable storage media that is capable of storing programinstructions or digital information.

The media used by persistent storage 305 may also be removable. Forexample, a removable hard drive may be used for persistent storage 305.Other examples include optical and magnetic disks, thumb drives, andsmart cards that are inserted into a drive for transfer onto anothercomputer readable storage medium that is also part of persistent storage305.

Communications unit 307, in these examples, provides for communicationswith other data processing systems or devices. In these examples,communications unit 307 includes one or more network interface cards.Communications unit 307 may provide communications through the use ofeither or both physical and wireless communications links. Programinstructions and data used to practice embodiments of the presentinvention may be downloaded to persistent storage 305 throughcommunications unit 307.

I/O interface(s) 306 allows for input and output of data with otherdevices that may be connected to each computer system. For example, I/Ointerface 306 may provide a connection to external devices 308 such as akeyboard, keypad, a touch screen, and/or some other suitable inputdevice. External devices 308 can also include portable computer readablestorage media such as, for example, thumb drives, portable optical ormagnetic disks, and memory cards. Software and data used to practiceembodiments of the present invention can be stored on such portablecomputer readable storage media and can be loaded onto persistentstorage 305 through I/O interface(s) 306. I/O interface(s) 306 alsoconnect to display 309.

Display 309 provides a mechanism to display data to a user and may be,for example, a computer monitor.

The programs described herein are identified based upon the applicationfor which they are implemented in a specific embodiment of theinvention. However, it should be appreciated that any particular programnomenclature herein is used merely for convenience, and thus theinvention should not be limited to use solely in any specificapplication identified and/or implied by such nomenclature.

The present invention may be a system, a method, and/or a computerprogram product at any possible technical detail level of integration.The computer program product may include a computer readable storagemedium (or media) having computer readable program instructions thereonfor causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that canretain and store instructions for use by an instruction executiondevice. The computer readable storage medium may be, for example, but isnot limited to, an electronic storage device, a magnetic storage device,an optical storage device, an electromagnetic storage device, asemiconductor storage device, or any suitable combination of theforegoing. A non-exhaustive list of more specific examples of thecomputer readable storage medium includes the following: a portablecomputer diskette, a hard disk, a random access memory (RAM), aread-only memory (ROM), an erasable programmable read-only memory (EPROMor Flash memory), a static random access memory (SRAM), a portablecompact disc read-only memory (CD-ROM), a digital versatile disk (DVD),a memory stick, a floppy disk, a mechanically encoded device such aspunch-cards or raised structures in a groove having instructionsrecorded thereon, and any suitable combination of the foregoing. Acomputer readable storage medium, as used herein, is not to be construedas being transitory signals per se, such as radio waves or other freelypropagating electromagnetic waves, electromagnetic waves propagatingthrough a waveguide or other transmission media (e.g., light pulsespassing through a fiber-optic cable), or electrical signals transmittedthrough a wire.

Computer readable program instructions described herein can bedownloaded to respective computing/processing devices from a computerreadable storage medium or to an external computer or external storagedevice via a network, for example, the Internet, a local area network, awide area network and/or a wireless network. The network may comprisecopper transmission cables, optical transmission fibers, wirelesstransmission, routers, firewalls, switches, gateway computers and/oredge servers. A network adapter card or network interface in eachcomputing/processing device receives computer readable programinstructions from the network and forwards the computer readable programinstructions for storage in a computer readable storage medium withinthe respective computing/processing device.

Computer readable program instructions for carrying out operations ofthe present invention may be assembler instructions,instruction-set-architecture (ISA) instructions, machine instructions,machine dependent instructions, microcode, firmware instructions,state-setting data, configuration data for integrated circuitry, oreither source code or object code written in any combination of one ormore programming languages, including an object oriented programminglanguage such as Smalltalk, C++, or the like, and procedural programminglanguages, such as the “C” programming language or similar programminglanguages. The computer readable program instructions may executeentirely on the user's computer, partly on the user's computer, as astand-alone software package, partly on the user's computer and partlyon a remote computer or entirely on the remote computer or server. Inthe latter scenario, the remote computer may be connected to the user'scomputer through any type of network, including a local area network(LAN) or a wide area network (WAN), or the connection may be made to anexternal computer (for example, through the Internet using an InternetService Provider). In some embodiments, electronic circuitry including,for example, programmable logic circuitry, field-programmable gatearrays (FPGA), or programmable logic arrays (PLA) may execute thecomputer readable program instructions by utilizing state information ofthe computer readable program instructions to personalize the electroniccircuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems), and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer readable program instructions.

These computer readable program instructions may be provided to aprocessor of a general purpose computer, special purpose computer, orother programmable data processing apparatus to produce a machine, suchthat the instructions, which execute via the processor of the computeror other programmable data processing apparatus, create means forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks. These computer readable program instructionsmay also be stored in a computer readable storage medium that can directa computer, a programmable data processing apparatus, and/or otherdevices to function in a particular manner, such that the computerreadable storage medium having instructions stored therein comprises anarticle of manufacture including instructions which implement aspects ofthe function/act specified in the flowchart and/or block diagram blockor blocks.

The computer readable program instructions may also be loaded onto acomputer, other programmable data processing apparatus, or other deviceto cause a series of operational blocks to be performed on the computer,other programmable apparatus or other device to produce a computerimplemented process, such that the instructions which execute on thecomputer, other programmable apparatus, or other device implement thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present invention. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof instructions, which comprises one or more executable instructions forimplementing the specified logical function(s). In some alternativeimplementations, the functions noted in the blocks may occur out of theorder noted in the Figures. For example, two blocks shown in successionmay, in fact, be executed substantially concurrently, or the blocks maysometimes be executed in the reverse order, depending upon thefunctionality involved. It will also be noted that each block of theblock diagrams and/or flowchart illustration, and combinations of blocksin the block diagrams and/or flowchart illustration, can be implementedby special purpose hardware-based systems that perform the specifiedfunctions or acts or carry out combinations of special purpose hardwareand computer instructions. The descriptions of the various embodimentsof the present invention have been presented for purposes ofillustration, but are not intended to be exhaustive or limited to theembodiments disclosed. Many modifications and variations will beapparent to those of ordinary skill in the art without departing fromthe scope and spirit of the invention. The terminology used herein waschosen to best explain the principles of the embodiment, the practicalapplication or technical improvement over technologies found in themarketplace, or to enable others of ordinary skill in the art tounderstand the embodiments disclosed herein.

Embodiments of the present invention may also be delivered as part of aservice engagement with a client corporation, nonprofit organization,government entity, internal organizational structure, or the like. Theseembodiments may include configuring a computer system to perform, anddeploying software, hardware, and web services that implement, some orall of the methods described herein. These embodiments may also includeanalyzing the client's operations, creating recommendations responsiveto the analysis, building systems that implement portions of therecommendations, integrating the systems into existing processes andinfrastructure, metering use of the systems, allocating expenses tousers of the systems, and billing for use of the systems.

1. A method comprising: receiving environment information correspondingto a user, wherein the environment information comprises information byusing an AR (Augmented Reality) device, selected from the groupconsisting of: traffic density during a commute of the user to asurgical area setting, a commute time of the user to the surgical areasetting, and weather at the surgical area setting; simulating, inaugmented reality, a target environment based on the environmentinformation, wherein the target environment corresponds to the surgicalarea; receiving physiological information corresponding to the user,wherein the physiological information comprises information selectedfrom the group consisting of: brain wave information, heart rate, breathrate, internal body temperature, and blood pressure; determining acognitive state of the user based on the physiological information,wherein a target cognitive state is stress; responsive to determiningthat the cognitive state of the user is not the target cognitive state,modifying, in augmented reality, the target environment by using an ARdevice to achieve the target cognitive state, wherein modifying thetarget environment comprises: simulating, in augmented reality, objectsor sounds in the target environment, and wherein objects include nurseavatars or patient avatars, and wherein sounds include surgical machinenoise; responsive to determining that the cognitive state of the user isthe target cognitive state, evaluating a user performance while the useris in the target cognitive state; and responsive to determining that themodification achieves the target cognitive state, storing themodification as an indicator of the cognitive state for the user.